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ABSTRACT 

To evaluate the efficacy of faculty seminars for 
writing across the curriculum given for the instructors at Robert 
Morris College (Pennsylvania), an examination of the instructors* 
verbal protocols was conducted. Eleven faculty members from a variety 
of disciplines who participated in the 45 hour-long seminars and 
eight who did not participate were tape recorded as they "planned out 
loud" a writing assignment for their respective courses. Participants 
were asked to describe their main concerns for teaching and learning 
as they devised the assignment. Analysis of the transcripts showed 
that teachers who participated in the writing across the curriculum 
seminars had a larger, more clearly defined repertoire of strategies 
for planning a variety of writing assignments appropriate for their 
courses. Additionally, unlike the nonparticipants, the participants 
used the writing assignments they planned to aid student learning, 
rather than just to test student knowledge. They also planned the 
assignment to help students solve particular problems, and integrated 
the assignment into what was being studied, so that students would 
see the relationship between writing and learning. The protocol 
analysis suggests that the writing seminars are an effective way to 
instruct teacners in the basics of content area writing instruction, 
and that they improve teacher attitudes and behavior concerning 
writing assignments. (Transcriptions of two teacher protocols are 
included, one from a seminar participant and the other from a 
nonparticipant • ) (JC) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

********* ******4r***********fr***** ****** ********* 



:RIC 



% 

' U.S. DCPAfrrUCNT OF EDUCATION 

Otftce of Educationar ROMarch and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
^ CENTER (ERIC) 

^^Sfhts document has been reproduced as 
> received from tne person or organization 

originating it 

□ Minor Changes have ^>een made to improve 
reproduction quatity 



Points of vi«w or opinic' « stated m this docu- 
ment do not necessarily represent official 
OERI position or policy 



O 

tn 

rH 
ITi 
00 
OJ 

o 

iU TEACHER PROTOCOLS: A NEW EVALUATION TOOL FOR 

WRITING ACROSS THE CURRICULUM PROGRAMS 

by 

JoAnn M. Sipple 
Robert Morris College 
Pittsburgh, PA 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

JoAnn M. Sipple 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CEN"! ER (ERIC) ' 



CONFERENCE ON COLLEGE COMPOSITION AND COMMUNICATION 

ATLANTA, 6A 

March 19-21, 1987 



K 

ERIC 



TEACHER PROTOCOLS: A NEW EVALUATION TOOL FOR 
WRITING ACROSS THE CURRICULUM PROGRAMS 



In their latest book Writing Across the Disciplines: Research 
into Practice (1986) Art Young and Toby Fulwiler make the point that 
when their program was already four years old, they had to reconsider 
what it was that they were actually able to evaluate. After they had 
practiced over a decade of writing across the curriculum at Michigan 
Technological University, Young and Fulwiler commented, "It now seems 
surprising how slowly it dawned on us that if we wanted to measure any 
effect produced directly by the program we should measure the effect 
closest to the actual treatment itself and not some effect several 
removes later. In other words, we should measure the impact on the 
faculty first and only later try to measure the impact on students" (Young 
& Fulwiler, p. 50). 

During those first four years Young and Fulwiler were conducting a 
series of faculty seminars to help faculty integrate write-to-learn 
assignments in all disciplines across the curriculum. They were also at 
that time collecting as much data as they could about student and faculty 
performances and at titudes. Yet, they found that what was actually 



measureable In the first few years was change In faculty attitudes toward 
writing; only later could they begin to measure improvement in student 
learning and writing. 

If the faculty must first plan to integrate multiple, various writing 
tasks in their courses before students can begin to use writing as a 
learning tool, it seems reasonable that faculty ought to be the first 
population to call upon for the institution's evaluation effort. Following 
the example of Young and Fulwiler, we at Robert Morris College (for our 
now three-year old program) have designed a multiple-measure evaluation 
collecting as much information as we can about faculty, student, and 
administration's respective attitudes toward writing. We have also sought 
to collect information about students' improved performance in the target 
courses of our writing across the curriculum program but have not yet 
been able to equate improved learning of a particular discipline with 
improved writing ability. Finally and most immediately, we have been 
systematically collecting data about faculty in their processes of planning 
and integrating writing into their particular course designs. 

it is this last area of data collection—faculty's planning and 
integration of writing tasks—that has led us to use protocol research 
methodology as an instrument for evaluating the successes and failures of 



our program. Protocol analysis is one of our many measures but, in my 
opinion, the most original and significant of all thus far. In fact, I know of 
no other writing across the curriculum program that has used teacher 
protocols as an evaluation measure in assessing an institution-wide 
program. 

As an evaluation tool, verbal protocols enable us to collect 
quantitative, raw data "in as hard a form as could be wished" (Ericsso ) & 
Simon, 1984, p.4). Talk-aloud protocols, made known outside studies of 
writing by Ericsson and Simon (1984), have been used to discover models 
of problem-solving processes. Flower and Hayes (1979) have used the 
same research method to discover cognitive models of the composing 
process. However, protocol research has not yet been explored as a means 
to discover the planning processes which teachers use as they design and 
integrate writing tasks in particular courses across the disciplines. 

In this paper, with some illustrations from the protocol transcriptions 
which we have collected from teachers in a variety of disciplines in the 
arts, sciences, and business at Robert Morris College, I would like to 
demonstrate that teacher protocols are not only a new but very effective 
evaluation tool for writing across the curriculum programs. Also, over the 
long term teacher protocols can provide valuable insights into model 
teacher planning processes as teachers design and integrate appropriate 



writing tasks discipline by discipline. 

For example, we have asked faculty to perform the following task for a 
talk-aloud protocol. That is, faculty talk aloud into a tape recorder and 
these tapes are later transcribed for analysis. 

THE TASK FOR THF TFArHFR IN A TAI K-AI nun PRnTnmi 

Devise one writing assignment for your course (name course). 
While you are devising the assignment, describe as fully as you 
can your main teaching/learning concerns. Talk aloud about 
what is going on in your mind while you are doing the task. 
Write the words for the assignment, which you would have typed 
hand to the student. 

Here are examples of the transcriptions of two teacher protocols from 
an upper-level course in management information systems. One of the 
teachers particpated in our forty-five semester hour long faculty 
seminars in writing across the curriculum while the other did not. 
Perhaps the most distinguishing feature between the two transcriptions is 
that the participant saw the writing astignment as an aid to learning— to 
achieving a specific course goal. i.e.. "distinguish between information and 
data." The participant thereby planned the writing assignment as an 
integral part of the course design to help the student achieve the course 
objective. The non-participant, on the other hand, saw the writing 
assignment as the finished product of the students' thinking— as a 
"persuasive communication to a superior" with al) the ccrrect features of 
spelling and grammar. ^ 
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S: <Ml^rightV',ah.r.J^;iTiiave"..t(?,des^gii 

itheh.I jieed.tOiJook.at the icours^/sc^euf and as I had before 
since this is the same course I have the same goal s. iS6l3^ (SS 

figtk^pOMjXsj^^ I don't know if you 

still have my goals from last tiine but if not we can zerox 
these. Ah - which I saidr given a list of items r be able to 
distinguish data from information. Name the attributes of 
information f state then in your own words. Those are^j;^ 
that's my operational objective for zM^T^ 
able to distinguished z^^^Vf^S^^^^^^P^ 
(inf o npaHon - and this is vtet I 
didn't do last time so that's why - it's brand new - I 
haven't done one for this particular goal. 

Ah - so I'll write that down - be able - to distinguish data 
from information - I just picked this one off the top of my 
head - I don't even know if I'll end up using it. My first 
thought is that what I want them to be able to do is -ah- 
appreciate the context of a user - if that makes any sense. 
I want them to picture the mind set of a user and therefore 
that before they can start deciding what's data versus 
what's information they need to -ah- identify a user and so 
I'm - I'm kjjid of thinking just broadly what kinds of -ah- 
what kind of things I would have them do. They're going to 
have to identify a user or have me provide it for them. I 

haven't decided that yet. Ah - they're going to have to 
-ah- decide vtet kinds of decisions - \ihat kinds - this 
doesn't look very narrative^ does it matter? I'll just say 
first students should identify use: then they should list 
what kinds of decisions and this I had trouble with last 
term - what kinds of decisions the user needs to make and 
what background the user has and vAat kinds of things are 
needed , 1 guess, what kinds of - I don't like to use the 
word information because that's part of the assignment - 
what kinds of -ah- well I'll just say - I don't know what to 
say - I'll leave it blank What kind of blank is needed for 
makLag decisions - making the decisions. 

And then, so that, I'm basically outlining my task the way 
I'm envisioning it right now. Yet what I have to give to 
the student should be -ah- some kind of a narrative 
description that would identify a particular user, would 
identify the kinds of things they do -ah- and kind of a 
general statement of what their goals are, what kinds of 
background they have and then the student would need to 
determine what kinds of information - 1*11 say it that way - 
is needed for making the decision. And then I v-ant them to 
be able to list or evaluate a list of itens that I give them 
to tell me whether they're data or whether they're 
information - whether they're - in this particular case. 
O And that would probably be my - and why. ^So I would askj* 



BEST >,VAII.ABLF, COF 



6 



|£m>aDation| I would envision this being done probably out 
of class so that we could discuss it in class. Ah - and 
-so in my own mind I've kind of outlined what I want to do 
but I haven't outline any of the specifics so that my next 
task would then be to identify a specific user that - that I 
thought they could particularly relate to. 

That's been one of my problems vhen I've done the course in 
the past is the students relate less to users than I expect 
them to. Last term I had a real problem with - with 
distinguishing decisions users make. Ah - and so - one idea 
that just cones to mind is maybe I want to think about a 
college registrar. Maybe they can appreicate what a college 
registrar might have to do. What kinds of decisions a 
college registrar would make. And -ah- that makes me think 
that it's probably better for me to ask them to come up with 
the narrative description of the user rather than me 
providing it. In the past I've done that so I'm kind of 
modifying my own statement as I started out and say - now I 
don't know why - it's just a feeling that I want them to 
express in as much detail as they can what kinds of - what 
kinds of decisions that are being made - that's an 
underlying goal of the whole course - the troi'ijle that they 
have. And then I can give them specific data in this case 
my big class a student - lots of data about a student like 
-ah- past grades, transcripts of data, financial 
information, names, addresses, all the stuff that I could 
think about about a student and -ah- I would ask them to 
evaluate whether or not the registrar let's say - I'm not 
sure that's a good one but some particular individual is 
going to - would find that as - that is what is infome.tion 
for that individual - what is required. 

Now, to actually design it it's probably going to take me a 
couple hours since I haven't done this one. I think the one 
I did for you before I had reasonably designed because I had 
examples there. This one is something I'm going to have to 
write -ah- so but that would be my general approach, but 
more the - 



ERIC 



BEST -A^'-^A|^>,PJ,F, irfy-y 



8 



TRANSCRIPTION OF A NON-PARTI CI PANT'S VERBAL PROTOCOL 




E: 



S: Al 

what he can do about it. in 

other words how he can correct the perceived problem in each 
case study. Ah - the main part of each case study - the 
body of each case study generally runs anyv^re from six to 
eight pages and the concluding part of it gives a number of 
questions that should be answered by each student. And then 
beyond that each student can add to it anything that he or 
she wants. But basically I want the student to write down 
the answer to each specific question after reading the 
particular case study twice. And the reason - the reason I 
like them to read it twice - I like then to read the study 
the first time and basically get a general idea of vtet the 
study is about and maybe get some thought patterns 
generating up in their mind. The second time I want them to 
read it in a more detailed fashion so that they can find the 
specifics of what may be wrong with the particular case. 

Could you sort of just keep talking, sort of talk out loud 
about what you're writing. 

So anyway after the second reading I feel that they're then 
able to start interacting. On some of these cases I permit 
them to interact with each other in discussing the case; 
however at the time that the students have - have generated 
enough oral exposure, then each student is required to write 
an opinion as to what the specific answers to the questions 
^t^the end of ,t±e easels tud^^^^ &HeH!H4y a're reqiiir^7 
J^f-JF^^e an,owrall^ 

^%.fathef..froni a global^ vjaTofmi: the main overall^ 
I concern ^is .and also what' should be, done to correct the 
t^j-t"ati^'-^ addition /.each student is required to cane'CpT 
J5*$ «>yi,i^enjlgt of alternate Solutions" to .the problen. J 
F>?..*^^?Wte .list is to" be perceived as the written' 
idocuirent .or,conrounication,jeport *to a superior ^Ithouoh the 
superior will often choose the desired alternates froiJi a 
list of solutions. I ask each student to select the 
alternate that they would prefer if they were the superior. 
.':I.al5o>sfc..each;5tpdent ;to present the case 'in' writing in*^ 
(^uch:a^way;as^to:convince the superior rthrough the' ^ 
jcomnunication process that 'the detailed solutions are ^ 
jcoraplete, coiprehensive/Tand .practical^ ' 

[ 'TheVtuaentsyillBeri' b^^ greaed'in^ccnijetitioH with each' 
1 .other, since no solution is necessarily better than another. 
I Therefore J look, for persuasive communication - 

BEST AVAILABLE COPY ^ 
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E: Try and talk about what you're thinking. 

S: Really - right now I'm just thinking to see if I may have 
missed anything • Ah - even though this is written more 
concisely than the way I am writing it now - obviously I may 
take hours to write this - in my mind I'm thinking about how 
I 'm going to present it verbally because students are never 
really carpletely satisfied with written instructions. They 
like more detailed verbal explanations and right now I'm 
thinking in my mind if I missed anything or how I'm going to 
explain these words to them verbally. 

Asfar as myjnain t^^aching/leaming concern -ah- obviously 

^itten cqnminicatlons akillaJcp^^Bol^^prob^ 
[ptes&nt ,it In Xpersuaisiye^^nne^'^'lll^ 
4tiBJf^Teport^,that the spellif^, graniiBr>'jSnd?pthejj-Beans^^^^^ 
i«^^^f ^ ,90^^f beyond '^"st a correct^TOlutiqri -perceived/ 
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These teacher verbal protocols provide both quantitative and 
qualitative data. (For a distinction between qualitative and quantitative 
data, see WitteandFaigley's ifj^j/i/j///?^ College Writing Programs 
1983). First, from a quantitative perspective and in the context of our 
experimental evaluation design, we have accumulated and analyzed data 
from eleven faculty (in the experimental group who underwent faculty 
seminars or writing across the curriculum) and from eight faculty in the 
control group (who did not participate in the writing across the curriculum 
faculty seminars but were teaching other sections of the same courses). 
The data show that teachers who particoated in the writing across the 
curri^lum seminars have a larger, more clearly defined repertoire of 
strategies for planning a variety of writing assignments appropriate for 
their courses. These data contrast sharply with those of the control group 
who had relatively few, if any, strategies to call upon for using writing as 
a learning tool. 

Out protocol analyst Nancy Penrose from Carnegie Mellon University 
was able to identify fourteen unanticipated features, which she measured 
in both the experimental and control groups of teachers, i e , in both the 
faculty participants and non-participants 1 1 986). Here are some 
contrasting results between the two groups for five of these features 

er|c u 
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CHART 1 



Features of Teachers' Planning Participant Non-Parfiripant 
of Writing Ta<ik«; 

The teacher realizes that creating 

an assignment Is a rhetorical task. 309S 13% 

The teacher is concerned that students 
see the purpose of t*-e writing. 

The teacher has thought about the task 
In concrete, operational terms— has 
considered the sub-tasks involved. 

The teacher is sensitive to students' 
abilities and acts on that informa- 
tion by modifying the writing task, 
providing extra guidance, etc. 



30?S 095 



100% QQ% 



90% 25% 



The teacher hopes that the writing 
assignment will help improve the 

students* writing skills. 0% 75% 



Remember that 75% of the non-participants were trying to improve 
student writing while none of the participants was trying to teach 
writing or improve writing skills. Rather the participants were trying to 
use writing in many ways to aid students in the learn mg process. The 
protocol transcriptions of the non-participants are particularly relevant 
here. 

I hope it the writing would help students feel less 
intimidated about science and math ... 1 think if it 
doesn't do that, it helps them to sit down and to 
write. I think it might help improve their writing 
skills a little. . . 
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. . .the chief executive officer . . .of Westinghouse 
pointed out that two of the key problems in hiring 
entry-level people today was their lack of writing 
skills and their lack of analytical skills. This writing 
task addresses both. . . 

From a qualitative perspective, we have also been concerned about the 

nature of the writing assignments which our faculty conceive as 

appropriate to assist students in their achievement of course objectives. 

We want to gain insights about whether our faculty are able to design 

writing assignments that genuinely function as aids to learning and not as 

just another test of students' knowledge. Again, some contrasting data: 

Those faculty who had participated clearly conceived of the nature, uses. 

and functions of writing differently from those who had not participated. 

In those Qualitative areas, these were the results in three of the eight 

discriminations which Richard Young of Carnegie Mellon and I were 

investigating: 



CHART 2 



Features of TparhPrs' Planning 
of Writing Tagk 



Participant 



Non-Particinant 



The teacher designs and uses the 
writing assignment to do more than 
test students' knowledge—to 
prmote student learning. 



100% 



50% 



The teacher plans the writing 
assignment to lead the student 
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toward solving a particular 
problem in achieving course 
objective(s). 
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80% 



30% 
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The teacher plans to integrate the 
writing assignment into the on- 
going learning process of the course. 90?? 50% 



The participants' protocol transcriptions are relevant here since 90?? of 
them planned writing assignments so that they would be integrated into 
the on-going learning processes of the course: 

This is to tie the . . . writing directly with the material 
that we're going over in class, whir . wouIg be relating 
management objectives and philosophy with actual prob- 
lems or situations that may be occurring within their work 
situation. 

So the concept of the journal with their written 
comments and evaluations will be important at the 
beginning for them to see the connection between 
the writing that they do in their journal and the 
speaking activities that we're going to have. 

Thus far these verbal protocols have provided us with the means to 
evaluate teachers' planning and designing of writing tasks discipline by 
discipline. These protocols have giVeh us some distinctions between 
participants and non-participants in terms of thirty-one measureable 
features of their writing assignments. And, as you can see from this 
sampling of data, the protocols have also enabled us to view the quality of 
what 's going on as teachers plan their courses— integrating their designs 
of writing tasks within their courses. 

Beyond their function as a powerful evaluation tool, teacher protocols 
serve yet another function: They have provided the base for many of our 

14 
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faculty to pursue their own research interests while at the same time 
opened the door to academic institutional research for Robert Morris 
College. Let me explain: 

These protocol data, which protocol researchers would argue provide 
incomparable hard evidence in evaluation, have helped us begin to imagine 
models of expert teacher planning processes for integrating meaningful 
writing tasl<s into the disicplines across our curriculum. The data which 
we continue to accumulate is allowing us to pursue two different sets of 
research questions: 1 ) What are the planning processes teachers use to 
construct write-to-leam tasks? Can we derive models of teachers' 
planning processes? 2) What are the writing processes students use to 
perform these write-to-leam tasks? Can we derive models of students' 
learning processes? 

Figure 1 illustrates the two sets of research questions tied to using 
protocol research methodology. In order to get at faculty learning 
processes, our protocol analysis allows us to examine whether faculty in 
the writing across the curriulum program designed writing assignments 
drawn from principles learned in the faculty seminars —to help students 
better achieve course goals. We are also planning to analyze student 
protocols in order to examine whether students participating In targeted 

er|c ^ ^ 
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FIGURE I 



TALK-ALOUD PROTOCOLS FOR 
PROGRAM EVALUATION AND INSTITUTIONAL RESEARCH 



TEACHER 
PROTOCOL 
DATA 




STUDENT 
PROTOCOL 
DATA 



FACULTY 
SEMINARS 



COURSE 
OBJECTIVES 



WRITE TO LEARN 
ASSIGNMENTS 



STUDENT 
WRITINO 



COURSE 
OBJECTIVES 



MODELS OF 
TEACHER 
PLANNING 
PROCESSES 



WRITE TO LEARN 
ASSIGNMENTS 



MODELS OF 
STUDENT 
LEARNING 
PROCESSES 



What are Uachar planning procasses? 
Can we derive models discipline by 
discipline? 



What are student learning processes 
Can we derive models discipline by 
discipline? 
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writing across the curriculum courses have developed more expert 
learning processes in performing the writing assignments to better 
achieve course goals. 

Perhaps in the not too distant future, we can use this data to generate 
expert systems in computer software to help faculty design and integrate 
writing tasks discipline by discipline. But for the moment, let's return to 
the reality of the data we hold. 

There are some conclusions that we have been able to draw about our 
program based on these data. First, in a formative way, the data we have 
been accumulating through protocols continues to help us monitor the 
seminars, classroom practices and other writing across the curriculum 
activities. Through these we have not only been able to discover what* s 
right and what's wrong with our program, but we have also been able to 
make necessary adjustments along the way. 

Second, there are a number of summative (evaluation) statements 
which the teacher protocols permit us to make: 

I. We can judge the success of our 45-hour intensive 

faculty seminars. For example, our protocol analyst Nancy 
Penrose offered these summative statements about our 
first set of data: 

The protocol/interview technique provides valuable 
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information for assessing the success of the faculty 
seminars. Overall, seminar particpants differed from 
non-participants on measures of attitude and teaching 
behavior. Participants typically view writing as a means 
for learning rather than as a testing device or an oppor- 
tunity for practicing writing skills. Their use of writing 
in the classroom reflects these attitudes. Participants are 
more likely than non-participants to develop assignments 
that further the learning objectives of their courses and 

that are integrated into the course structure The results 

of the present analyses indicate that the faculty seminars 
provide an effective means for communicating the fundamental 
principles of the writing across the curriculum program and for 
changing the way writing is used in courses at Robert Morris" 
(Penrose, 1986). 

2. The protocol analysis corrobrates the findings from our 
analyses of teacher attitude surveys. That is, our participants 
have changed their conception of writing assignments and their 
place in their course designs, and our faculty have acted on that 
information. 

3. The protocol analysis reveals that participating faculty know 
how to plan for and design writing tasks that aid learning and 
that further they know how to plan courses that accommodate 
write-to-Iearn tasks. 

4. The protocol analysis reveals that faculty participants view 
writing much differently from non-participants. 

5. The faculty's experience with protocols has shown them the 
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significance of their contributions in the institution's 
evaluation effort and has also provided them with a database 
to pursue collective and individual research efforts. 
Thus, the research methodology helps us get an idea of what the 
participants take away from the seminars and of how this training affects 
their work and attitudes. It >s important to note that such information is 
not readily available through other means such as surveys or classroom 
visits. 

In their 1986 text Writing Across the Discip fines, Young and 
Fulwiler explained why they kept pursuing their multiple-measure 
approach to evaluation. They say In their book, "Collect all the data 
possible because It simply gives us more inofrmation on which to base our 
overall assessment" (p.52). I heartily concur with their aporoach. And In 
addition I would recommend using teacher protocols in the context of 
multiple-measure evaluation. Teacher protocols specifically lend i^ight 
Into ways which teachers can more effectively integrate writing tasks 
within their course designs. And they can guide teachers in planning 
writing assignments into their courses so that students will more 
efficiently achieve their course objectives. 
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